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The effectiveness of learning paired associates from 
a computer-based drill strategy known as the progressive state drill 
was compared in this study with the effectiveness of having students 
use their own strategies with flashcards. Characteristics of the 
progressive state drill which make it a potentially effective drill 
and practice strategy for learning paired associates include; (1) the 
use of a small working pool of items; (2) increasing ratio review; 

(3) dynamic adjustment of the drill based on student performance; and 

(4) recordkeeping from session to session. The 96 high school 
sophomores who acted as subjects were placed into matched pairs on 
the basis of a pretest which measured their ability to remember 
word-number pairs, and one member of each pair was randomly assigned 
to the flashcard or microcomputer group. The treatment consisted of 
three 20-minuts sessions, i.e., one session each day for three 
consecutive days. Although the results failed to show any superiority 
on a posttest achievement measure for the progressive state drill 
over the flashcard approach, the microcomputer group demonstrated a 
significantly more positive attitude toward the instruction. A 
discussion of the implications of these findings for design and use 
of microcomputer drills such as the progressive state paradigm 
conclude the paper. Statistical analyses and a list of 26 references 
are appended. (MES) 
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Abstract 



Among the uses of the computer in education, drill and practice 
ranks as one of the most popular. Yet, in spite of the 
pervasiveness of these drills, relatively few studies have been 
done co compare the effectiveness of various computer-based drill 
strategies with each other or with other methods of practice. 
This study compares the effectiveness of learning paired 
associates from a computer-based drill strategy known as the 
progressive state drill with the effectiveness of having students 
use their own strategies with flashcards. The progressive state 
drill is a fairly sophisticated drill structure which includes 
several characteristics which make it appear to be a potentially 
effective drill and practice strategy for learning paired 
associates. Some of these characteristics include: use of a 
small working pool of items, increasing ratio review, dynamic 
adjustment of the drill based on student performance, and record 
keeping from session to session. Results, however, failed to 
show any superiority on a posttest achievement measure for the 
progressive state drill over the flashcard approach. There was a 
difference in attitude of the two groups toward the instruction, 
with the microcomputer group demonstrating a significantly more 
positive attitude. Implications for design and use of 
microcomputer drills such as the progressive state paradigm are 
discussed. 
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A Comparison of a 
Microcomputer Progressive State Drill and 
Flashcards for Learning Paired Associates 



Computer drills are becoming more and more popular. Various 
computer-based drill ?nd practice strategies have been described 
in the literature (Allesi and Trollip, 1985; Atkinson, 1974; 
Merrill & Salisbury, 1984; Salisbury, in press; Seigal & Misselt, 
1984). Usually these computer-based drill strategies are 
compared with, simple flashcard strategies to show their 
superiority. However, few studies have been done to compare 
these various strategies with one another or to compare the 
effect of learning from a particular computer-based drill 
strategy with the effect of having students use their own 
strategies with flashcards. 

The purpose of this study was to compare the effectiveness 
of a computer-based drill strategy known as the progressive state 
strategy with the effect of having students learn the same 
mate) ial using their own strat°gy with flashcards. The study 
examined the effect of the progressive state drill on student 
post-test performance and attitude toward the instruction. The 
post-test performance and attitude of a group of subjects who 
tried to learn the content by means of the progressive state 
drill were compared against the same measures of a group of 
subjects who tried to learn the content by using their own 
strategy with flashcards. 



Description of the Progressive State Drill Strategy 

The progressive state drill strategy includes several 
characteristics which make it potentially effective. Some of 
these are: (a) use of a small working pool of items, (b) an 
increasing ratio review schedule, (c} dynamic adjustment of the 
drill based on student performance, and (d) record keeping from 
session to session. The term progressive state is used to 
describe the drill since items are presented to the learner in a 
progressive sequence passing from one presentation state to 
another based on performance criteria for each state. Box 1 
shows a flowchart for a progressive state drill. In this drill 
items are presented in six different states. These are: 

1. Pretest state - used to determine if the learner 
already knows the item 
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2. Rehearsal state - presents item and response 
simultaneously on the screen for a brief moment to 
allow learner to associate them together. The learner 
is then asked to type in the correct response. 

3. Drill state - presents the item and prompts learner for 
correct response 

4. First Review strte - identical to how item was 
presented in drill state 

5. Second Review :tate - same as first review state 

6. Third Review state - same as first review state 



Insert Box 1 about here 



Each item in the drill has a state number associated with 
it. This state number indicates the current state of the item 
and determines the presentation state in which the item will be 
presented. When the learner begins a practice session any items 
left in the learner's working pool from the last session are 
transferred from the disk to the current working pool (Step 1). 
Review dates and state numbers corresponding to each of the items 
are also transferred from the disk. If the working pool is not 
full, then additional items are selected from the review pool if 
there are any "eview items which have review dates les. than or 
equal to today's date (Steps 3 and 4); otherwise additional items 
are selected from the new item pool (Step 5). After the working 
pool is shuffled (Step 6), the first item is selected and its 
state noted (Steps 7 and 8). If the item has just entered the 
drill as a new item, it will be presented as a pretest item 
(State 1). If the learner responds correctly to the item, it 
will be deleted from the system. Otherwise, its state will be 
updated to State 2. The value of the item counter (N) is then 
incremented, and the second item from the working pool is 
selected (Step 7 again). This item is presented in accordance 
with its specified state. After all the items in the current 
working pool have been presented once (N 7), the working pool 
is replenished and shuffled (Steps 1 - 6), and the item counter 
(N) is set back to 1 (Step 7). This process is repeated until 
the learner terminates the practice session. Note that after an 
item is presented in State 3 (drill), it is removed from the 
working pool and transferred into the review pool. The review 
pool is divided into three states to provide increasing ratio 
review. 

Simply stated, increasing ratio review oieans that new and 
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review items will be intermixed throughout the drill with the 
ratio of review to new items increasing as the drill progresses. 
When the learner first begins the drill, all items will be new 
items. As the learner masters items, these become review items 
and are reintroduced systematically into the drill based on time 
elapsed since the learner was last presented that item. Toward 
the end of the drill most of the items will be review items with 
only a few new items being introduced. 

Drills which are structured in this way are very useful for 
the purpose of skill maintenance in addition to initial learning. 
Learners can work on the drill initially to master the content 
and then they can continue to use the drill from time to time to 
review and re-check mastery. Typically, once a person has 
attained a high level of accuracv and precision on a skill, it 
can be maintained at that level over a long period of time with 
only a small amount of practice at regular intervals. 

In order to construct a drill with increasing ratio review 
it is necessary that each item in the review pool have a review 
date associated with it. This review date indicates when the 
item is next to be reviewed. When the learner begins a practice 
session, any items left in that learner's working pool from the 
last session are transferred from the disk to the current working 
pool. The working pool is shuffled and the items in the working 
pool are presented to the learner. If the learner gives a 
correct response to an item the item goes into the review pool. 
If an incorrect response is given the item remains in the working 
pool. Steps 3-5 of Box 1 show how the working pool is 
replenished. The procedure first checks to see if there are any 
items in the review pool which are ready for review (Steps 3 and 
4). An item is ready for review when its review date is equal to 
or less than the current date. O^erwise, the replacement item 
is a new item from the item pool (Step 5). 

Hypothesis 

The presence of the desirable instructional characteristics 
of the progressive state drill mentioned above (small working 
pool, increasing ratio review, dynamic adjustment of the drill 
based on student performance, and record keeping from session to 
session) give reason to believe that students using this drill 
might perform better on a post-test than students using their own 
strategies with flashcards. However, there is also a theoretical 
basis to suggest the flashcard group might learn better than 
students using the progressive state drill. The basis for this 
arises from cognitive learning research which suggests that 
students learn verbal information by paying attention to the 
"meaning" uf the material and organizing structural 
representations of the material in their minds (Anderson, 1980; 
Salisbury, 1984; Salisbury, Richards, & Klein, 1985). To promote 
the formation of these structural representations, students 
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should be able to identify important relationship m sequences, or 
groups present in the material. The progressive state drill 
paradigm, like most other computer-based drill paradigms, 
presents only one item at a time to the student, making it 
difficult for the student to form his or her own groupings or 
sequences. In contrast, with flashcards, students can lay out 
several flashcards on the table to form groups and they can group 
them any way they wish. Pilot research done by the experimenters 
suggested that with learners who have fairly well developed study 
strategies, the flexibility of flashcards provides an advantage 
over a structured computer-based drill such as the progressive 
state drill. Since a theoretical base existed to suggest the 
possible superiority of either group, a two-tailed statistical 
test was chosen. The hypothesis (stated in the null form) was 
that there was no statistically significant difference in 
learning between the PSD group and the flashcard group. A second 
null hypothesis was that there was no statistically significant 
difference in attitude toward the instruction between the two 
groups. 



Design 

This study utilized a statistical methodology known as 
sequential analysis (King & Roblyer, 1984; Weed & Bradley, 1971). 
In sequential analysis the sample size for an experiment is not 
specified in che research design. Instead, certain decisions are 
made by the investigator prior to the collection of data. The 
characteristics of these data as they are obtained and analyzed 
determine the point at which the number of observations is 
sufficient for the experiment to be terminated. 

In a sequential analysis study, a set of observations is 
obtained and the data from this first set of observations is 
analyzed. Based on the results of this and any prior 
observations, one of three decisions is made: (a) accept the 
null hypotheses, (b) accept an alternative hypothesis, (c) or 
obtain more observations. The principal advantage of sequential 
analysis over designs which specify a fixed sample size is its 
greater efficiency. Since, in sequential analysis, data are 
analyzed as they are obtained, it is often the case that 
decisions can be reached with fifty percent or fewer observations 
than would be possible with other types of research designs. 



Method 

The task consisted of learninq 100 word-number pairs which 
were unfamiliar to the subjects, samples of word-number pairs 
are 1 - HAT, 10 - TOES, 30 - HEAD. The complete set of 100 items 
is shown in Figure 1. Subjects were high school sophomores from 
the Developmental 
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Insert Figure 1 about here 



Research School at Florida State University. Ninety-six 
potential subjects were pretested using the Object-Number Test 
developed by Educational Testing Service (1963) as a measure of 
their ability to remember word-number pairs of the type that were 
to be employed in the study. Subjects were placed into matched- 
pairs based on their pretest scores and one member of each pair 
v/as randomly assigned to either the group using flashcards or to 
the microcomputer group. 

The treatment consisted of three twenty minute sessions, one 
session each day for three consecutive days. Subjects in the 
flashcard group were allowed to use the flashcards in any manner 
they wished but were required to work alone. Subjects in the 
microcomputer group used the progressive state drill to learn the 
same material. Due to space limitations, the study had to be 
carried out over a period of several weeks, with an equal number 
of subjects assigned to the two treatments each week. This 
process was continued until, under the rules of sequential 
analysis, enough observations had been obtained to accept or 
reject the two null hypotheses. 



Results 

Post-test Data . As mentioned in the design section, in a 
sequential analysis, after the data from one set cf observations 
have' been analyzed, one of three decisions is made: (a) accept 
the null hypothesis, (b) accept the alternative hypothesis, (c) 
or obtain more observations. The point at which one arrives at 
one of these three alternatives will be influenced by the design 
parameters specified in advanced by the researchers. For this 
study, we set alpha at .05, beta at .20, and specified an effect 
size (ES) of three-fourths of a standard deviation, meaning that 
we were interested in detecting any difference between groups 
that was larger than 3/4 S.D. (or about 9 items on the post- 
test). Two groups each containing nine pairs of subjects were 
tested before the decision was made to stop. Table 1 shows the 
signed differences between the pairs. In group one, enly eight 
differences are shown because the scores of one pair were tied 
and no information concerning the relative effectiveness of the 
two treatments could be obtained from them (see Wilcox on Signed 
Rank Test, 1945). A negative sign indicates that the score of a 
flashcard subject was greater than that of his/her paired PSD 
counterpart. 



Progressive Drill 



Insert Table 1 about here 



Since the test statistic indicated that the difference between the 
two groups was not significant at the parameter levels specified, the 
decision was to accept the null hypothesis and to conclude thai there 
was no significant difference in the performance of the progressive 
state drill group and the flashcard qroup on the posttest. Because 
the effect size of interest (3/4 S.D.) was chosen in advance (the 
effect size being the minimal amount of effect which would be of 
practical importance to detect) this result can also be interpreted as 
indicating that there is no difference that is of practical importance 
between the two groups. 

Attitude Data . Table 2 shows the signed differences between 
pairs on the attitude measure. This time the test statistic 
indicated that the difference between the groups was significant 
and the decision was made to accept the alternative 



Insert Table 2 about here 



hypothesis and conclude that there was a significant difference 
in attitude between the progressive state drill group and the 
flashcard group. The raw data on the attitude measure indicated 
the difference was in the direction of the progressive state 
drill group with that group exhibiting a more positive attitude 
toward the instruction. Again, because the effect size of 
interest was specified in advance, this result can be interpreted 
to be a significant difference in statistical terms as well as in 
practical terms. Means and standard deviations for the 
progressive st ite drill group and for the flashcard group are 
presented in \ .e traditional fashion in Table 3. 



Summary and Discussion 



The research question explored by this study was whether 
there would be a significant difference in post-test performance 
or in attitude toward the instruction between a group of subjects 
that learned paired-associations using a progressive state drill 
on a microcomputer and a group which used their own strategy with 
flashcard. The results of this study show no significant 
difference in post-test performance between subjects using 
flashcards and those using the progressive state drill. However, 
there was a difference in attitude of the two groups toward the 
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instruction with the microcomputer group demonstrating a 
significantly more positive attitude. Some conclusions and 
implications for the design of practice drills which can be drawn 
from the study are given below. 

First, it might be concluded that students in this age group 
have fairly sophisticated strategies for learning from 
flashcards. The subjects in this study tended to use fairly 
sophisticated practice and review strategies with the flashcards. 
For example, most subjects separated the 100 flashcards into 
separate piles for learning and most reviewed flashcards which 
they had learned in a fairly systematic manner. 

Second, even though the subjects in the flashcard group 
exhibited a less positive attitude towards the instruction, they 
learned as many word-number pairs as the subjects in the PSD 
group. This may add to the body of research which suggests that 
students do not learn the most from the type of instruction they 
claim to like the best (Clark, 1982; Clark, 1983). 

This study also suggests that designers of computer-based 
drills more seriously cons;der the implications of cognitive 
research on learning verbal information. There is a great deal 
of literature to suggest that the more meaningful the to be 
learned material is, the easier it will be learned (Anderson, 
1976; Wanner, 1968; Pompi and Lachman, 1967). Because students 
can remember meaningful information better than meaningless 
information, instruction should seek to make material as 
meaningful as possible. This can be done by providing images 
which relate things together or by emphasizing networks or 
relationships inherent in the content (Bower, 1970b; Dansereau, 
1978; Weinstein, 1978, 1982; Vaughn, 1981) and also thrcjgh the 
use of acronyms, mediators, and mnemonics. Other memory 
techniques such as link and loci (associating material to be 
learned with spatially organized objects and places) have been 
shown to help impose some arbitrary meaning on otherwise 
meaningless material (Bower, 1970a; Bower, 1970c; Gilbert, 1978; 
Higbee, 1977). 

Computer drills which present single items to the learner 
one at a time are generally intended, not for initial learning, 
but for practice of material to which the student has already 
been exposed. Computer drills are generally intended to be used 
in conjunction with some other means of instruction (computer 
tutorial, narrative presentation, textbook, etc.) which presents 
the material initially to the learner. The initial learning of 
the material should allow the student to utilize some of the 
techniques mentioned in the previous paragraph. Perhaps 
flashcards provided the subjects in this study more opportunity 
to form groups and study relationships among the items, than did 
the PSD. 
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The result showing that the subjects using the PSD exhibited 
a more positive attitude toward the instruction is of practical 
importance. What this means is that even though students might 
learn as well using flashcards as they would if they used the PSD 
on a microcomputer, it is unlikely that they would be motivated 
to stick with the task. The computer drill is significantly more 
motivating and engaging. In this study, the flashcard group 
functioned in a supervised, structured environment and this 
undoubtedly contributed to the on-task behavior on these 
subjects. 

Finally, some questions can be raised concerning the 
generalizability results of this study. Since the PSD is 
designed to be used over a long period of time, it is possible 
that th design characteristics of the PSD (increasing ratio 
review, keeping track of working pool from session to session) 
did not have time to demonstrate their effect during the duration 
of the experiment. Different results would perhaps be obtained 
were the subjects to use the PSD for a longer period of time. 

It should also be pointed out that the results of this s';udy 
should not be generalized to different age groups. Younger 
students undoubtedly would use less sophisticated strategies with 
flashcards and a comparison between them and the PSD would 
possibly render very different results. 

We would like to ad<J a note of caution. This study should 
not be viewed as a member of that genre of studies known as 
"media comparison 11 studies. The study is not a comparison of 
nncroconiputers 2nd flashcards. It does not seek to determine 
which medium is more effective for learning in general, or for 
learning paired-associates, specifically. Rather, it is a 
comparison of one particular drn strategy (the PSD) with 
students 1 own flashcard strategics. Different results might be 
obtained for different strategy comparisons. 

Although the results of the study showed "no significant 
difference" in post-test performance between the two groups, we 
feel that this study provides a valuable contribution to the 
literature on computer-based instruction in that it is one of the 
experimental studies in this area in which the researchers 
specified in advance the alph* level ( of Type 1 error), the 
effect size of interest, and „/e statistical power (1-beta) 
desired to detect a difference if it did exist. Only when these 
levels are specified in advance can a sample size be assured that 
is adequate to detect a difference between the groups. 

Also, the methodology known as sequential analysis is a 
valuable alternative methodology for researchers in the area of 
computer-based instruction. Sequential analysis allows the 
researcher to avoid having to predetermine the sample size. In 
most cases, a decision can be reached with fewer observations 
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than would be possible with other types of research designs. 
Sequential analysis also requires the investigator to make 
certain decisions prior to conducting the experiment regarding 
t, . conditions under which the null hypothesis can reasonably be 
regarded as true. Recent secondary analyses of the statistical 
power of research studies in the behavioral sciences (Daly & 
Hexamer, 1983) indicate that in the great majority of studies 
power is quite low, (i.e. type two error rates are largel. When 
using sequential analysis, the researcher is required to consider 
power a priori thus assurinn that the sample size is large enough 
to detect a difference betwev the groups. 

Computer drill and practice programs are widely viewed as 
potential instructional tools for teaching. More studies should 
be done to compare various computer-based drill and practice 
paradigms with one another and with non-computer-based 
strategies. 
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Author Note 



The specific sequential analytic technique used in this 
study was the Sequential One-Sample (Paired Samples) Configural 
Signed Rank Test (CSRT) originated by Weed and Bradley (1971). 
The design parameters required by this test are alpha, beta, and 
the effect size (ES) one wishes to be able to detect. For this 
experiment, these parameters were set as follows: alpha = .05, 
beta = .20, ES = three-fourths of a standard deviation. In this 
test the differences between the scores of the matched pairs in 
each group are compared and rank ordered in terms of their 
absolute magnitudes. The probability of the particular 
configuration of signed ranks is computed under a hypothesis of 
no difference, Hq, and also under an alternate hypothesis, Hj, in 
which a difference in favor of one group of 3/4 S.D. or more is 
postulated. Data are then collected until the researcher reaches 
the point at which Hq or Hj can be accepted. In the current 
study, a two-tailed version of the CSRT was desired so the 
probability ratio was modified to test the alternative hypothesis 
that the ES was + 3/4 S.D. 

The authors would like to thank F. J. King, Ken Brewer, and 
David Paulsen for their help with the design of this study and in 
the use of sequential analysis and Patricia Tolbert and Diane 
Fryman for their assistance in running the study. 
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Table 1 

Signed Achievement Differences Between Pairs of Students in the Two 



Groups: Posttest 



Rank Ordered Configuration 
Differences Between Pairs of Signed Ranks 



Group 1 


Group 2 


Group 1 


Group 


-21 


34 


-X 


X 


-18 


28 


-X 


X 


-13 


-17 


-X 


-X 


9 


-16 


X 


-X 


-7 


16 


-X 


X 


-6 


-8 


-X 


-X 


-5 


-8 


-X 


-X 


3 


5 


X 


X 




-3 




-X 



Note 1: A negative sign indicates that the score of the flashcard 

subject in that pair was greater than his/her PSD counterpart. 
In group one, only eight differences are shown because the 
scores of one pair were tied and no information concerning the 
relative effectiveness of the two treatments could be derived 
from them. 

Note 2: The rejection region for the test statistics is based on alpha 
=.05, B = .20, and ES = 3/4 S.D. 
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Table 2 



S nned Differences Between Pairs of Students jrn the Two Groups: 
Attitude Measure 



Rank Ordered Configuration 
Differences Between Pairs of Signed Ranks 



Group 1 


Group 2 


Group 1 


Group 


45 


39 


X 


X 


44 


37 


X 


X 


30 


24 


X 


X 


29 


-22 


X 


-X 


24 


19 


X 


X 


12 


-15 


X 


-X 


-11 


-14 
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Note 1: A negative sign indicates that the score of the flashcard 
subject in that pair was greater than his/her counterpart. 



Note 2: The rejection region for the test statistics is based on elpha 
=.05, B = .20, and ES = 3/4 S.D. 



Table 3 

Means and Standard Deviations for the PSD Group and the Flashcard Group 

Posttest Attitude 
PSD Flashcards PSD Flashcards 
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